A Focused Backpropagation Algorithm for Temporal Pattern Recognition

Time is at th e heart of many pat tern recognition t asks, e.g., speech recognit ion . However, connectionis t learning algorithms to date are not well suited for dealing with tim e-varying input patterns. This paper introduces a specialized connectionist architecture and corre sponding specialization of the backpropagation learnin g algori thm th at opera tes efficiently on temporal sequences . The key feature of t he archit ecture is a layer of self-connecte d hidden units that integrate their curre nt value with th e new input at each time ste p to construct a static represent ation of the temporal input sequence . Thi s architecture avoids two deficiencies found in other models of sequence recognition: first , i t reduces the difficulty of temporal credit assignm ent by focusing th e backpropagated err or signal; second, it eliminates the need for a buffer to hold th e input sequence and/or intermediat e activity levels. The lat ter prop erty is due to the fact th at during th e forward (activation) phase, incremental activity tra ces can be locally compute d that hold all information necessar y for backpropagation in time . It is argued tha t thi s architecture should scale better t han conventional recurrent architectures wit h respect to sequenc e length . The architecture has been used to implement a temporal version of Rumelhart and McClelland's verb past-tense model [1]. The hidden units learn to beh ave something like Rumelhart and McClelland 's "Wickelphones," a rich and flexibl e representation of temporal information.